Image auditing method and system

ABSTRACT

An image auditing method and system may be provided. Video surveillance equipment may be operated in coordination with at least one digital computer across network architecture. The video surveillance equipment may provide a computer with footage of a transactions occurring in a target area. The footage may be filtered into at least one image of a transaction. The image may be categorized based on at least one transaction characteristic. The transaction data may be compared to transaction records from a point of sale and analyzed for any discrepancies.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/071,578 filed Sep. 29, 2014, and entitled Video analytics forloss control, the entire contents of which are hereby incorporated byreference.

BACKGROUND

The point of sale may be the time and place where a retail transactionis completed, cancelled, or partially completed. A point of saleterminal may be a device that records, organizes, and implements therelevant transaction data of a specific point of sale. Point of saledevices may utilize customized hardware and software tailored to aspecific requirement and purpose. One example may be the earlyelectronic cash registers at restaurants in the 1970's that may haveallowed employees to input a customer's order by numeric keys whiledisplaying the customer order on a display device for verification andfeedback. Modern systems have improved upon the basic foundationalbuilding blocks of point of sale terminals to offer additionalcustomizations and features. Modern point of sale devices may beenhanced by bar code readers, pin pad displays, and reporting features.Point of sale devices have greatly improved the accounting and inventoryrecord keeping of retail businesses.

Video surveillance may be the use of video cameras to transmit a videorecording to a specific location or to store video surveillance footagein a specific location. Video Surveillance Equipment may havehistorically consisted of cameras physically linked via hard cables totransmit video recordings to recording devices, display devices, orboth. Video surveillance may often be employed when human surveillanceis not feasible. Video surveillance may have been used in retail storesto monitor customer and employee activities.

Cloud computing may be a form of information technology managementconsisting of a client side computing device, a server side computingdevice, and a network architecture that allows the client and serverside computing devices to communicate. A client side computing devicemay access a software platform hosted by the server side computingdevice across a web-browser. The software platform may be accessed ondemand as software as a service licensing and delivery model in whichthe software is licensed on a subscription basis and is centrally hostedby the server side computing device.

Presently, there may exist a desire for a practical application of videosurveillance technology, and cloud computing technologies, to be appliedat a point of sale. The combination of these technologies maysubstantially improve reporting of sales and inventory.

SUMMARY

According to an exemplary embodiment, a method of auditing transactionswith video surveillance may be provided. The method may includeproviding video surveillance equipment at a target area, configuring theequipment to record at least one transaction at the target area, andallowing the equipment to record and store at least one image or videofile of the at least one recorded transaction in a storage component. Atleast one processor may further be provided and configured tocommunicate with the storage component. The at least one processor maybe configured to filter the at least one image or video file to onlyimages relevant to a transaction. The at least one processor may furtherbe configured to categorize and tag the images according to transactioncharacteristics. The at least one processor may finally create atransaction data file according to the categorized images for comparisonwith recorded point of sale transaction data.

According to another exemplary embodiment, an image audit system may beprovided. The image audit system may include video surveillanceequipment configured to record at least one image or video file of atransaction. The image audit system may further include at least onecomputer for receiving the at least one image or video file from thevideo surveillance equipment. The computer may filter the image or videofiles based on recognition of a transaction into at least one relevantimage and categorize and tag the at least one relevant image based on atleast one transaction characteristic. The image audit system may furtherinclude network architecture coupling the video surveillance equipmentand the at least one computer.

According to yet another exemplary embodiment, a non-transitory computerreadable medium for creating a transaction record for auditing purposesmay be provided. The non-transitory computer readable medium may includeinstructions to be executed on a processor. The instructions may causevideo or image footage from video surveillance equipment to be receivedover a network. The instructions may further cause the footage to befiltered into at least one image relevant or contextual to atransaction. This may be determined through at least one of timegrouping, background averaging, background subtraction, and imageredundancy analysis. The instructions may further cause the at least oneimage to be categorized and tagged based on at least one of location andmotion of an item in relation to a threshold between a server side andcustomer side of a target area, image recognition, and probabilityindexing. The instructions may finally cause the tagged relevant orcontextual images of a transaction to be saved as a transaction datafile for potential comparison to transaction records.

BRIEF DESCRIPTION OF THE FIGURES

Advantages of embodiments of the present invention will be apparent fromthe following detailed description of the exemplary embodiments. Thefollowing detailed description should be considered in conjunction withthe accompanying figures in which:

FIG. 1 may show an exemplary embodiment of a video surveillance system;

FIG. 2 may show an exemplary flow chart of the steps of an exemplaryimage audit process;

FIG. 3 may show an exemplary flow chart of the steps of an exemplaryimage audit process;

FIG. 4 may show an exemplary flow chart of the steps of an exemplaryimage audit process;

FIG. 5 may show the components of an exemplary image audit system;

FIG. 6 may show an overview of the steps an image audit system mayperform;

FIG. 7 may show the relationship of the components of an exemplary imageaudit system;

FIG. 8 may show exemplary characteristics or attributes recorded by thecomponents image audit system;

FIG. 9 may show the relationship of the components of an exemplary imageaudit system; and

FIG. 10 may show the steps of an exemplary image audit process.

DETAILED DESCRIPTION

Aspects of the invention are disclosed in the following description andrelated drawings directed to specific embodiments of the invention.Alternate embodiments may be devised without departing from the spiritor the scope of the invention. Additionally, well-known elements ofexemplary embodiments of the invention will not be described in detailor will be omitted so as not to obscure the relevant details of theinvention. Further, to facilitate an understanding of the descriptiondiscussion of several terms used herein follows.

As used herein, the word “exemplary” means “serving as an example,instance or illustration.” The embodiments described herein are notlimiting, but rather are exemplary only. It should be understood thatthe described embodiments are not necessarily to be construed aspreferred or advantageous over other embodiments. Moreover, the terms“embodiments of the invention”, “embodiments” or “invention” do notrequire that all embodiments of the invention include the discussedfeature, advantage or mode of operation.

Further, many of the embodiments described herein may be described interms of sequences of actions to be performed by, for example, elementsof a computing device. It should be recognized by those skilled in theart that the various sequence of actions described herein can beperformed by specific circuits (e.g., application specific integratedcircuits (ASICs)) and/or by program instructions executed by at leastone processor. Additionally, the sequence of actions described hereincan be embodied entirely within any form of computer-readable storagemedium such that execution of the sequence of actions enables theprocessor to perform the functionality described herein. Thus, thevarious aspects of the present invention may be embodied in a number ofdifferent forms, all of which have been contemplated to be within thescope of the claimed subject matter. In addition, for each of theembodiments described herein, the corresponding form of any suchembodiments may be described herein as, for example, “a computerconfigured to” perform the described action.

According to at least one exemplary embodiment, an image audit systemmay be disclosed. The image audit system may include a computing devicethat may be operated at a physical location such as a retail outlet,warehouse, commercial setting, university setting, office setting, orother physical location that may function as a point of sale. Inalternative exemplary embodiments, the computing device may optionallybe hosted off-site. In some exemplary embodiments, a combination ofon-site and off-site computers may be utilized. The computing device maybe a desktop computer, server, tablet, smart phone, or other similarlydesigned device. The image audit system may further include asurveillance system configured to record footage of a transaction. Thefootage may be recorded as video or image recording files, which mayconsist of video footage from multiple transactions or a singletransaction. A transaction may be the sale of an item, movement ofinventory, the opening or closing of an enclosed space, or servingrelated transactions. The surveillance system may have the capability toprovide the recording file to the computing device.

Now referring to exemplary FIG. 1, an exemplary surveillance system 100may include a camera sensor component. A camera may record footage tolocal storage 102 attached to the camera, to a local network device, ordirectly to a cloud server 104. An exemplary surveillance systemcomputer 106 may optionally have a dual or quad core processor. Inembodiments having multi-core processors, one core may be in charge ofmotion detection. This may allow for hyper sensitivity of motion aroundan interest area. If properly optimized, the same core may optionally beused to store the result of the motion detection in a specifiedlocation. Alternatively, a separate core may be used to store the resultof the motion detection in a specified location, such as a direct bufferto FTP. In an exemplary embodiment, motion may trigger the system tosave footage in the camera buffer 102 and pass the footage to apersistence-component 108. The persistence-component 108 may negotiate anetwork, send the footage, and confirm reception. If transmission issuccessful, the footage may be deleted from the buffer 102. Iftransmission fails, the send may be repeated at a set interval until thebuffer can be dumped. The send retries may optionally be performedfirst-in, first-out. If remote-storage is not available and the localcamera buffer is full, the system may start overwriting. The overwritingmay optionally be performed first-in, first out. Any remaining cores maybe used to perform standard surveillance camera functionality, such as,but not limited to, power, networking, time synchronization, alerts, andother functionality as would be understood by a person having ordinaryskill in the art. Particular additional functionality may includescheduled call home functionality with auto-configuration. This may beperformed at predetermined intervals, such as daily. A camera may, forexample, send a scheduled message daily, which may be used as amonitoring ping to determine that the location is functioning properlyand to ensure firmware is up-to-date. In an exemplary embodiment, themessage may optionally be implemented as an HTTP request with a customHTTP response. The HTTP response may let the camera know if it needs todownload new settings or firmware. Configurable settings on a camera'sweb interface may be automatically set from an administration backend.

Exemplary camera sensors may include HD surveillance camera sensors. Insome exemplary embodiments, these may include thermal or infraredsensors. Thermal or infrared functionality may be utilized for low lightsituations or for determining object characteristics, as discussedbelow. Cameras may be any suitable camera as would be understood by aperson having ordinary skill in the art. In some exemplary embodiments,cameras may have a minimal profile to reduce intrusiveness andfacilitate installation. In some further exemplary embodiments, thecamera may be a camera system on chip, which may be connected to powerin a strategic location and camera sensors may be connected via a cable.

Now referring to exemplary FIGS. 2-3, at least one processor implementedin the surveillance system, on-site computer, or remote server mayautomatically filter or cull the footage into an image, or series ofimages, related to each transaction. The filter component in anexemplary embodiment may include a multi-step inference engine which mayresult in the reduction of non-relevant footage. The audit system may bestrategically configured to facilitate the filtering of footage. Thesystem may be configured by defining area of interest points. Forexample, this may include a counter area or point of sale. Once camerasare installed, a field of vision reference may be established. A fieldof vision reference may be established by providing an irregularquadrilateral, which may be used to calculate perspective of items. Forexample, two similar items on different ends of a counter or targetarea, one closer and one farther from a camera, may appear as havingdifferent sizes. A blob size percent may be established, which may serveas a threshold for observation. Anything below a pre-determined blobsize percent of the interest points may be ignored. An employee sidedistinction may be configured. This may allow the filtering module totarget motion initiated from an employee side of a target area. Anemployee/customer side distinction threshold may be configured such thatthe system distinguishes movement originated close to either side. Theconfiguration may further establish a background learning rate. Thebackground learning rate may set an amount of images the filteringmodule may use to make an inference, as would be understood by a personhaving ordinary skill in the art. The desired rate may vary withdifferent environments, such as between fast paced locations and slowerlocations. Pre and post contextual images may also be configured. Theseimages may not include recognition of a transaction, but may be used toprovide context.

Once configured, an exemplary system may operate as follows.Surveillance footage may be accessed by a filter component 200 of acomputer device. The images may be time grouped 202. Voids of time maybe detected to determine distinct transactions. Footage files may besaved with a file name or metadata indicating a full date and time ofthe capture. This may be used to determine subsequent images. It may beinferred that even if images are similar in nature, too many secondsbetween the capture of the footage may indicate the footage should betreated as distinct transactions. Analysis of the footage may beperformed to average or establish a baseline moving background 204. Forexample, all moving pixels for all images may be analyzed such that theycreate a composite moving background, which every image may then becompared to. If there is no change to the background for a large portionof footage, such as 200 images, and then there is a background changethat is consistent over another large portion of footage, such as 400images, two distinct composite backgrounds may be formed. The 200 imageportion will then be compared to the composite background created fromthose 200 images and the 400 image portion will be compared to thecomposite background created from the 400 image portion of footage. Thismay provide context necessary for monitoring transactions.

Using the averaging results, the backgrounds and similarities of imagesor footage portions may be subtracted 206 to determine which images maybe relevant for auditing purposes. The prior analysis may result inlogical contextual groups with averages in subsequently calculatedimages. The reference moving background within a contextual group may besubtracted from each image. Once the background has been removed, theresulting image may be compared to the pre-established blob sizepercentage. If the resulting image is less than the configured blob sizepercentage, the whole image may be discarded, not deleted, as irrelevantfor subsequent calculations. Discarded images may be retained forpotential future use. The first image of a group may always beconsidered relevant and therefore may always be included in calculationsdespite the result of its background calculations.

The relevancy of the images may further be refined based on imageredundancy analysis 208. The images within each resultant group from theprior steps of analysis may be compared for pixel variation. Theanalysis may determine if there is enough pixel variation between twoimages to warrant keeping both. The pixel variation may be apre-configured threshold value. The first image of a group may remainrelevant regardless of this analysis. In an exemplary embodiment, theimage redundancy analysis may be performed as follows. Image 1 may becompared to Image 2, if Image 2 is not sufficiently similar, Image 2 maybe discarded and Image 1 may then be compared with the next set ofimages until there is sufficient change in an image. The image havingsufficient change may then become the reference image and the analysismay be repeated based on the new reference image. Each image notsufficiently different may be discarded, but not deleted, except for thefirst image of a group 210.

The relevant and contextual images may then be moved 212 to an analyticscomponent from the filter component 200. Relevant images may include theimages resultant of the previous steps and which also fall within theconfigured threshold distance to an employee side of a target area.Contextual images may include images immediately before and immediatelyafter relevant images. Contextual images may have been previouslydiscarded as irrelevant during the previous calculations and analysis.Once the analytics component confirms reception of the relevant andcontextual images, the full footage file may be deleted and left in arunning state until the next run of filtering.

In the image analytics and categorization component 220, the resultantimages from the filtering engine may be received 222. Blobs, as definedabove, may be extracted from the resultant images 224. The blobs mayundergo desired processing to facilitate analysis, such as theapplication of a smoothing algorithm. Detected shapes in the images maybe compared to a database including known shape data or other indicatingcharacteristics to determine a type of item or transaction shown in theimage. Images may be grouped into informal groups for analysis. Aninformal group may include every relevant image with its relatedcontextual non-relevant images selected by the filtering component. Inan exemplary embodiment, the image analytics and categorizationcomponent may analyze the images 226 as follows. Any pixel differencesdetected in the filtering component for an image may be compared to animage database. Common shapes associated with items may be detectableand categorized with a similarity probability index, which may be basedon how similar the image is to the known database data. In scenarioswhere other elements may interfere with the detection of a common shape.An example may include a human element when an image shows the actualpassing of an item being sold. In such an instance, the interferingelement, such as the human element, may be detected through imagerecognition, as would be understood by a person having ordinary skill inthe art. The interfering element may then be substantially eliminatedand the remaining item shown in the image may be compared forcategorization. Each categorized image may be given a probability index228, which may indicate the probability of a match between the detectedimage and the known item database records. Each blob within an image maybe given a probability index and may be ordered by probability index230. The probability index may be related to a general type of item,such as a bottle of beer or bottle of wine. In some exemplaryembodiments, a human may review 232 images and enter a determinationinto the system. The determination and the imagery upon which thedetermination was based may be added to the known reference database tobe considered by the categorization component in subsequent runs. Theresulting categorized data may establish a transaction data set 234,which may subsequently be used.

Now referring to exemplary FIG. 4, the auditing system may receive pointof sale data from a customer in addition to the transaction data set242, which may then be compared to the transaction data set by amatching component 240. The matching component may receive the point ofsale data from a customer and compare it to resultant images andtransaction data within the same date and time range from the analyticscomponent. In an exemplary embodiment, point of sale data may be sent tothe matching component over a network utilizing a POS interface. Pointof sale data may include transaction characteristics, such as, but notlimited to, POS item name, transaction date and time, POS revenuecenter, employee name or ID, and other relevant characteristics as wouldbe understood by a person having ordinary skill in the art. It may benecessary for the POS transaction data to include an item identifier,such as a name, and a temporal identifier, such as the date and time. Inembodiments where an item identifier is a name or numbers, these may bespecific to a particular enterprise or may have distinct meanings fordifferent enterprises. For example, a screwdriver may be a drink atrestaurants and a tool at a hardware store. Therefore, an equivalencytable may be generated to translate an enterprise's item identificationsto a matching component acceptable identifier 244.

The matching component may compare the POS transaction data with thetransaction image data 246. The transactions that are accounted for inboth data sets may be eliminated. The comparison may utilize enterprisespecific configurable standard operating procedures. For example, thismay be used to regulate how long an employee has to register atransaction in a POS. Image transaction records that have not beeneliminated due to a matching POS transaction record may indicatepotential loss due to a failure to meet the standard operatingprocedure. This may include, for example, failure to process atransaction in a given time or at all. Relevant and contextual imagesfor unmatched transactions may be flagged 248. The results may becompiled in a report 250, which may be sent to a customer enterprise.Reports may optionally be sent periodically, in real-time, or may beavailable on-demand. For example, a customer may be able to access areport via a web interface.

The categorization and comparison of transactions may include in depthcharacteristics of the transaction, such as the time, place, location,item type, item quantity, item weight, item color, item shape, and otherrelevant transaction characteristics such as the person overseeing thetransaction. On the auditing system side, these characteristics may bedetermined by the surveillance system/computer, an auditing server, or ahuman auditor. If an item, person, or other transaction characteristicis not discernible from the footage, the surrounding footage may bescrolled through to account for the missing transaction characteristic.The categorization may be dependent upon the physical properties of thetime, items, and persons involved in the transaction. The image filesmay be tagged based on the categorization. At least some of the specificcharacteristics of a transaction may also optionally be recorded at aPOS and included in POS transaction data. These recorded characteristicsmay be utilized during the comparison. Known characteristics of an itemmay be accounted for by the matching component. For example, similar tothe equivalency data, known item characteristics may be entered into thesystem, such that the known characteristic data may be tagged to a POStransaction identifying an item type. The matching component may checkfor recognition of similar transactions, including time, item type,quantity, and other characteristics discernible from the point of saletransaction data files. The matching component may flag and reportanomalies between the image audit files and the POS transaction datafiles. For example, if a transaction recorded in the image audit filesis not accounted for in the POS transaction files, it may be flagged andreported. Each flagged transaction may optionally undergo additionalreview, including review of surrounding transactions.

Referring now to exemplary FIG. 5, the components of an exemplaryembodiment of an image audit system may be disclosed. A computer 505,video surveillance system 501, and a point of sale device 503 may beshown. The video surveillance system 501 may include a video camera,night vision camera, infrared camera, thermal-infrared camera, motionsensing camera, a single frame camera, a multi frame camera, or anyother similarly designed device as would be understood by a personhaving ordinary skill in the art. The video surveillance system 501 mayconsist of multiple video cameras in varying locations. The videosurveillance system 501 may be configured to record video footage of atransaction relevant to a point of sale device. The video surveillancesystem 501 may record the location of the video footage. The locationmay be further specified by a global positioning system or altimeter.The location may be further specified to a building by floor zone anddepartment. The video footage may be archived as a video database. Thevideo database may record attributes such as but not limited to time andplace. The video database may consist of a portion of the video footageor all of the video footage.

The video surveillance system may further include network architecture507 to facilitate communication among a computer device 505,surveillance system 501, and point of sale device 503. The networkarchitecture 507 may be a local area network or it may be a networkcapable of accessing the World Wide Web. Alternatively, communicationmay be provided manually by a flash drive, tape, DVD, or other similarlysituated devices understood by a person having ordinary skill in theart. The computer 505 may have a network adapter to access the internetby wireless or wired connection. The computer 505 may be implemented inor configured to receive data from the surveillance system 501. In someembodiments, the computer 505 may optionally be an on-site device, maybe a remote server, or a combination of both. The computer 505 mayfilter the video database into single or multiple images of atransaction. The computer 505 may categorize an image and assign certaindesired characteristics. Exemplary inputs may include: the time, place,point of sale ID, person overseeing the transaction, item type, itemnumber, item color, item shape, item contents, item value, and otherinputs known to a person having ordinary skill in the art of point ofsale devices and inputs. The categorization may be programmable suchthat any number of variable inputs may be used for specificapplications.

The point of sale device 503 may be a tablet, digital computer, cashregister, touch screen, or application specific point of sale device.The point of sale device 503 may store the details of a transaction as atransaction data file. The transaction data file may store the locationof a sale. The location may be further specified by a global positioningsystem or altimeter. The location may be further specified to a buildingby floor zone and department. The transaction data file may betransferred to the computer 505 across network architecture 507. Thecomputer 505 may compare the transaction data file to the resultantinventory characteristics data file. For example, the comparison mayconsist of a conditional operation in which the item type and itemquantity of the resultant inventory characteristics data file arecompared against the item type and item quantity of the transaction datafile. In some instances, a transaction data file may not exist at all.If the inputs of the inventory characteristics data file do not match atransaction data file, the computer 505 may report the discrepancies anddifferences. The report may be sent by email, SMS, Bluetooth or othercommunicatory capabilities, as would be understood by a person havingordinary skill in the art. As an example, the report may be sent by atext message or email, in real time, to a manager or security officer'smobile device that a discrepancy has been detected. The report functionsmay occur automatically or they may be produced at the request of anoperator. The report may be accessed and viewed on a display device 521.The display device 521 may be an additional component of the digitalcomputer 505 or it may be a monitor at an alternate location on or offsite. FIG. 6 may show an exemplary flow chart of the above describedauditing process 600.

Referring generally to FIG. 7, an exemplary embodiment of the networkarchitecture of an image audit system may be disclosed. The networkarchitecture 702 may refer to the physical technological elements of alocal area network such as Ethernet cables, network controller devices,coaxial connections, fiber optic connections, and other physicalelectronic wiring that may enjoin digital devices. The networkarchitecture 702 may further refer to the physical technologicalelements such as a wireless controller, wireless access point, wirelessrepeater, wireless range extender, and other physical electronictechnological elements that may be understood by a person havingordinary skill in the art to enjoin digital devices wirelessly. Thenetwork architecture 702 may also refer to an external utility operatednetwork infrastructure such as Ethernet cables, fiber optic cables,coaxial cables, 3G, 4G, LTE, and other physical technological elementsthat a person having ordinary skill in the art would understand aninternet service providing utility company may utilize. The networkarchitecture 702 may use any unique combination of the aforementionedphysical technological elements. The network architecture 702 may beused to enjoin devices of an exemplary video auditing system such as butnot limited to: the video surveillance system 704, a server side digitalcomputer 706, a point of sale device 708, and optionally a client sidedigital computer 710.

Referring generally to FIG. 8 an exemplary embodiment of thecharacteristics or attributes recorded by a point of sale device 802 anda video surveillance device 804 of an image audit system may bedisclosed. A point of sale device 802 may input characteristics thatcompose the varying elements of a transaction data file 818. The pointof sale device 802 may input characteristics as would be understood to aperson having ordinary skill in the art. Exemplary input characteristicsmay be the time of sale 806, inventory item and quantity 808, and othercharacteristics 810. Other characteristics may be recorded at a point ofsale or tagged based on item type. Other characteristics may include therelative size, color, shape, price, weight, temperature, and otherdesired characteristics. The video surveillance system 804 may recordfootage of a transaction. The video surveillance system 804 may recordfootage of characteristics that compose the varying elements of a videodatabase 820. The footage may be named according to, or have metadatashowing, the time of the sale 812 and the location of the sale. Thefootage may show other transaction characteristics 816 such as an imageof a customer, a storekeeper, a clerk, a security officer, and otherphysical attributes of the inventory item such as the relative size,color, shape, price, and weight. The video surveillance equipment maycapture more video footage than is needed. A video database 820 mayconsist of segments of video footage on an as needed basis or it mayconsist of the entire bulk video footage aggregated over a time period.The footage may further be divided into distinct images. A computer oroptionally a human auditor may filter or cull the bulk video footage tothe video footage or image that may be relevant to a point of saletransaction, as described in detail above. In some embodiments, thevideo surveillance device 804, may be configured to only record videofootage when a point of sale device is active or activated.

Referring generally to FIG. 9, an exemplary embodiment of client andserver side computing devices of an image audit system may be disclosed.A point of sale device 902 may transfer a transaction data file 906across network architecture 910. The transaction data file 906 may betransferred to a client side computing device 912, a server sidecomputing device 914, or both. A video surveillance system 904 maytransfer a video database 908 across network architecture 910. The videodatabase may be transferred to a client side computing device 912, aserver side computing device 914, or both. The server side computingdevice 914, may perform additional manipulation of the video database908, the transaction data file 906, or both. In some alternativelyexemplary embodiments, the client side computing device 912 mayoptionally be a component of the video surveillance system 904. In yetfurther exemplary embodiments, there may optionally be no client sidecomputing device 912. The server side computing device 914 or clientside computing device 912 may filter the video database into a discreteimage or a series of discrete images correlating to the time and place atransaction may have occurred. The point of sale device 902 and thevideo surveillance device 904 may be time synchronized such that thetime recordation of a point of sale transaction and the time recordationof the video footage of a transaction would be substantially identical.The synchronization may be used as a basis for the comparison oftransaction records. The transaction data file may also be filtered sothat the as needed information may be utilized more efficiently.

The video database, or the resultant image or images as previouslyexplained, may be categorized according to attributes of thetransaction. Attributes may include inventory characteristics of themerchandise or goods of the point of sale transaction or the attributesmay include other characteristics surrounding the person or personsinvolved with the transaction. The inventory characteristics may reportthe serving or selling of items that may not have transactionidentification for a point of sale device. The attributes may includecustomer information, storekeeper information, clerk information,security officer on duty information, and other physical attributes ofthe inventory item such as the relative size, color, shape, price, andweight. The categorization may include the categorization of atransaction data file. The transaction data file may be beneficial tocategorize so that system resources can be allocated effectively andefficiently. For example, a transaction data file may containinformation that a transaction was voided, a return was made, anunorthodox quantity of items were sold, a discount was applied, astorekeeper or salesperson oversaw the sale, or other occurrences thatwould likely warrant further investigation. The system may be programmedto categorize automatically or with the assistance of a human operator.

The categorization may create an inventory characteristics data file.The system may be custom programmed to recognize and categorizeobservations based on specific industry requirements, custom situations,or by a standard operating procedure. Further to the above description,the filtering and categorizing of footage may optionally include imagerecognition, facial recognition, pattern recognition, digital watermarkrecognition, three-dimensional image recognition, and other imagerecognition configurations that would be understood by a person havingordinary skill in the art of image recognition. Pattern recognition, mayinclude the categorization of a label to a given input value. Patternrecognition may include algorithms that generally aim to provide areasonable attribute or categorization of all possible attributes and toperform a “most likely” match of the inputs. Pattern recognition maytake into account the statistical variation of the “most likely”determination and apprise the input of a probability assessment inaddition to classifying the input. The server side computing device 914may compare the transaction data file 906 and the inventorycharacteristics data file, as referenced above. Additional comparisonmay be performed when the standard operating procedure may have not beenadhered to or other occurrences that may warrant additional review.

The server side computing device 914 may report the detailed comparison.The report may include instances in which the standard operatingprocedure was breached, modified, or unknown and non-calculable issuesmay have occurred. As an exemplary hypothetical, the report may includeinstances in which the inventory characteristic data file indicated thatthree items were sold and the transaction data file indicated that twoitems were sold. In this instance, the report would flag the transactionfor further review. The report may additionally compile the footage fromthe video database 908 and the original point of sale transaction datafile 906, for additional review. The report may optionally go throughadditional rounds of verification in which statistical probabilities areassigned to the categorization and comparison. The report may betransferred or accessed through the network architecture 910 to aclient. The client may receive or access the report as desired, such asthrough a client side computing device, a mobile device of a storemanager, clerk, security officer, etc. In some embodiments, the reportmay be sent or accessed in real time, directly after the transaction iscompleted or upon the initial detection of an anomaly. The real timereporting may apprise store clerks and security officers at the exit ofa building to perform an additional verification of the physical goods.The report may contain alerting features which may, in some instances,warrant a report and an ensuing immediate alert while other instancesmay not. The report may be allocated to specific recipients that may bedependent upon the severity of the alert or the accuracy of thestatistical analysis.

Referring generally to FIG. 10, an exemplary image auditing process maybe shown. Video surveillance equipment may record a transaction 1002.Footage of a transaction or transactions may be stored for processing1004. The footage may be filtered to include only relevant portions oftransactions 1006, as described in further detail above. In someembodiments, the footage may be duplicated so that an unedited originalcopy may be referenced at a later point in time. The transactions shownin the filtered footage may then be categorized 1008, as detailed above.The categorization of a transaction may consist of substantially similarinformation to that which would be found in a point of sale transactiondata file. Next, the inventory characteristic data file may be compared1010 to the transaction data file of a point of sale device. Next, anyanomalies between inventory characteristics and the transaction datafile may be reported 1012. The report may occur in real time,periodically, or on-demand. The report may be stored and accessed at alater time. Next, the optional step of verification of the reportedanomalies 1014 may occur. The verification 1014 may rely on a desiredlevel of statistical certainty as discussed previously. The verification1014 may alternatively be set to no statistical certainty required inwhich all anomalies may be reported 1012. Finally, the aforementionedsteps may be repeated as desired 1016.

In some further potential applications, recorded characteristics of atransaction may be compared with pre-configured standard operatingprocedures set by a client. For example, thermal or color characteristicdata may be used to determine an item quality. Therefore, if an item ina transaction does not meet a pre-configured standard operatingprocedure indicating a threshold temperature, for example, it may beflagged. Standard operating procedures may further include dress,timeliness, and other quality control aspects of a transaction. Otherdesired item characteristics or procedures may be similarly audited, aswould be understood by a person having ordinary skill in the art.

The foregoing description and accompanying figures illustrate theprinciples, preferred embodiments and modes of operation of theinvention. However, the invention should not be construed as beinglimited to the particular embodiments discussed above. Additionalvariations of the embodiments discussed above will be appreciated bythose skilled in the art.

Therefore, the above-described embodiments should be regarded asillustrative rather than restrictive. Accordingly, it should beappreciated that variations to those embodiments can be made by thoseskilled in the art without departing from the scope of the invention asdefined by the following claims.

What is claimed is:
 1. A method of auditing transactions with videosurveillance, comprising: providing video surveillance equipment at apoint of sale; configuring the video surveillance equipment to recordand store at least one image or video file comprising at least onerecorded transaction in a storage component; providing at least oneprocessor configured to communicate with the storage component, whereinthe at least one processor filters the at least one image or video fileinto a files comprising only images relevant or contextual to atransaction by grouping the original footage based on time andbackground, averaging the background for a group, subtracting thebackground, analyzing the resultant pixels for compliance with athreshold, discarding redundant images based on pixel analysis, andretrieving discarded images that are contextual to a transaction;configuring the at least one processor to categorize and tag the imagesrelevant and contextual to a transaction according to transactioncharacteristics; and configuring the at least one processor to create atransaction data file according to the categorized images for comparisonwith recorded point of sale transaction data.
 2. The method of claim 1,wherein images relevant to a transaction are further determined bymonitoring the location and motion of an item in reference to a serverside and customer side of a threshold.
 3. The method of claim 2, whereinimages contextual to a transaction comprise images that are first in agroup, last in a group, or immediately preceding or following an imagerelevant to a transaction.
 4. The method of claim 2, wherein theprocessor is further configured to use image and pattern recognition inthe filtering and categorizing of images.
 5. The method of claim 4,further comprising configuring the filtering and categorizing of imagesto assign a probability index based on image recognition results.
 6. Themethod of claim 1, further comprising allowing a point of saletransaction record to be compared to the transaction data file by amatching component implemented through the at least one processor. 7.The method of claim 6, further comprising allowing the matchingcomponent to flag any discrepancies between the point of saletransaction record and the transaction data file and configuring thematching component to generate a report of any discrepancies.
 8. Animage audit system comprising: video surveillance equipment configuredto record at least one image or video file of a transaction; at leastone computer configured to receive the at least one image or video filefrom the video surveillance equipment, filter the at least one image orvideo file based on recognition of a transaction into at least onerelevant image and categorize and tag the at least one relevant imagebased on at least one transaction characteristic; and at least onenetwork architecture coupling the video surveillance equipment and theat least one computer.
 9. The system of claim 8, further comprising atleast one point of sale device configured to record and transmittransaction data to the at least one computer.
 10. The system of claim9, wherein the at least one computer further comprises a matchingcomponent configured to compare transaction data from the at least onetagged relevant image to transaction data from the point of sale device.11. The system of claim 10, wherein the at least one computer isconfigured to flag and report any discrepancies between the transactiondata from the at least one tagged relevant image and the transactiondata from the point of sale device.
 12. The system of claim 11, whereinthe discrepancy comprises an tagged relevant image indicating atransaction and no corresponding transaction registered on a point ofsale device in accordance with a pre-configured standard operatingprocedure.
 13. A non-transitory computer readable medium for creating atransaction record for auditing purposes, comprising instructions storedthereon, that when executed on a processor, perform the steps of:receiving video or image footage from video surveillance equipment overa network; filtering the footage into at least one image relevant orcontextual to a transaction, as determined through at least one oflocation and motion of an item in relation to a threshold between aserver side and customer side of a target area, time grouping,background averaging, background subtraction, and image redundancyanalysis; categorizing and tagging the at least one image based on atleast one of image recognition and probability indexing; saving thetagged relevant or contextual images of a transaction as a transactiondata file for potential comparison to transaction records.
 14. Thenon-transitory computer readable medium for creating a transactionrecord for auditing purposes of claim 13, further comprisinginstructions stored thereon, that when executed on a processor, performthe steps of receiving transaction records over a network from a pointof sale device, and comparing the transaction records from the point ofsale device with the transaction data file.
 15. The non-transitorycomputer readable medium for creating a transaction record for auditingpurposes of claim 13, wherein background averaging further comprisesorganizing images into groups having similar backgrounds and comparingpixels of images within the same group against a composite movingbackground for that group.
 16. The non-transitory computer readablemedium for creating a transaction record for auditing purposes of claim13, wherein an image contextual to a transaction comprises imagessurrounding images showing a transaction.